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Abstract 

Background: Traditional scientific workflow platforms usually run individual experiments with little evaluation and 
analysis of performance as required by automated experimentation in which scientists are being allowed to access 
numerous applicable workflows rather than being committed to a single one. Experimental protocols and data 
under a peer-to-peer environment could potentially be shared freely without any single point of authority to 
dictate how experiments should be run. In such environment it is necessary to have mechanisms by which each 
individual scientist (peer) can assess, locally, how he or she wants to be involved with others in experiments. This 
study aims to implement and demonstrate simple peer ranking under the OpenKnowledge peer-to-peer 
infrastructure by both simulated and real-world bioinformatics experiments involving multi-agent interactions. 

Methods: A simulated experiment environment with a peer ranking capability was specified by the Lightweight 
Coordination Calculus (LCC) and automatically executed under the OpenKnowledge infrastructure. The peers such 
as MS/MS protein identification services (including web-enabled and independent programs) were made accessible 
as OpenKnowledge Components (OKCs) for automated execution as peers in the experiments. The performance of 
the peers in these automated experiments was monitored and evaluated by simple peer ranking algorithms. 

Results: Peer ranking experiments with simulated peers exhibited characteristic behaviours, e.g., power law effect 
(a few dominant peers dominate), similar to that observed in the traditional Web. Real-world experiments were run 
using an interaction model in LCC involving two different types of MS/MS protein identification peers, viz., peptide 
fragment fingerprinting (PFF) and de novo sequencing with another peer ranking algorithm simply based on 
counting the successful and failed runs. This study demonstrated a novel integration and useful evaluation of 
specific proteomic peers and found MASCOT to be a dominant peer as judged by peer ranking. 

Conclusion: The simulated and real-world experiments in the present study demonstrated that the 
OpenKnowledge infrastructure with peer ranking capability can serve as an evaluative environment for automated 
experimentation. 



Background 

This study demonstrates the use of peer ranking of 
multi-agents (peers) on a peer-to-peer scientific experi- 
mentation platform. Experiments with simulated peers 
were conducted to show that sophisticated peer ranking 
algorithms are applicable to the OpenKnowledge infra- 
structure. Experimental demonstration of peer ranking, 
albeit with a simple algorithm, of actual computational 
services for tandem mass spectrometry (MS/MS) protein 
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identification were still useful in monitoring the perfor- 
mance of peers during their interactions. The rest of 
this background section will describe the concepts and 
technologies applicable to this study. 

OpenKnowledge infrastructure 

It is commonplace for bioinformaticians to use experi- 
mental protocols that coordinate programs with the aim 
of finding novel results. It is uncommon, however, for 
these protocols to be viewed as a key aspect of experi- 
ment design. Instead, the protocols normally are viewed 
as part of the experimental infrastructure; they specify 
how to run the experiment but do not directly define 
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the experiment. For this reason, the protocols are sel- 
dom made sharable even if they are reusable. The Open- 
Knowledge system provides a method for describing 
experimental protocols in a high level declarative lan- 
guage called Lightweight Coordination Calculus (LCC) 
so that the specification of a protocol can be indepen- 
dent of the means used to automate it and a general- 
purpose mechanism (the OpenKnowledge system) 
through which these protocols may be shared within 
peer groups (Figure 1). 

In Figure 1, frame 1 (top left) depicts the peer to peer 
network of computers, some being operated by humans 
others being automated programs (such as database ser- 
vices). Protocols are designed by humans and contribu- 
ted to the peer to peer network by an advertisement 
mechanism (using tags to give a simple keyword-based 
characterisation of the protocol). In frame 1, peer pi 
contributes an experimental protocol with three inter- 
acting roles (depicted as black circles in the diagram). 
Other peers acquire experimental protocols by posting a 
query to the peer network and receiving a list of proto- 
cols with (approximately) matching tags. This is analo- 
gous to searching for a Web page except that routing 
through the network replaces a centralised system of 
query servers. In frame 2, peer p2, acquires the experi- 
mental protocol contributed by pi. If a peer finds an 
experiment that is useful to it then it may offer it for 
subscription by appropriate collaborating peers. If the 
choice of peers is already determined (e.g. when the 
experiment involves coordinating a known collection of 
Web services) then there is no need for subscription by 



peers other than the one initiating the protocol. If, how- 
ever, the initiator does not know which peers might 
undertake some of the roles in the experimental proto- 
col then it may offer it for subscription (and in turn it 
will be offered subscribers as these become available). In 
frame 3, peer p2 offers the experimental protocol for 
subscription and peers p3 and p4 subscribe to appropri- 
ate roles. Once a protocol is fully subscribed it can be 
used (via the OpenKnowledge interpreter) to coordinate 
the interaction between subscribed peers. This can be 
done by maintaining the coordination on a server (the 
traditional method); or by distributing the roles to the 
peers (a "pure" peer to peer method); or by selecting 
one of the peer group to act as host. The latter method 
is used in the current OpenKnowledge system but the 
experimental protocol is defined independently of this 
choice (so the protocols don't have to change if the 
infrastructure for coordination changes). In frame 4, 
peer p4 is hosting the coordination between p2, p3 and 
itself. 

Crucial to this view of experimentation (Figure 1) is 
the idea that a single, formal specification describes the 
abstract experiment design and is also capable of being 
automated in a simple, uniform way to perform actual 
experiments. This basic idea is not new-the idea of 
executable specifications (explained in the context of 
LCC in [1]) permeates declarative programming (giving 
a basis for logic programming and functional program- 
ming) while in bioinformatics there exist systems such 
as Taverna [2] that provide design tools for scientific 
workflow specifications and, in the MyExperiment [3] 
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Figure 1 Sharing experimental protocols in OpenKnowledge system. 
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system, allow these to be tagged with meta-data (analo- 
gous to frame 1 in Figure 1) while also supplying infra- 
structure for executing these workflows (analogous to 
frame 4 in Figure 1). Where OpenKnowledge differs 
from Taverna (and similar bioinformatics workflow and 
pipeline systems) is that we generalise these principles 
to a peer to peer setting and for generic (not only bioin- 
formatics) applications such as emergency response (see 
the OpenKnowledge Web site at http://www.openk.org 
for example applications and scenarios). Specifically: 

OpenKnowledge is not specifically targeted at work- 
flow. Although it can be used to implement workflows, 
its core language (LCC, described later) is a generic, 
declarative language for describing coordination between 
processes that synchronise through message passing. In 
later sections of this paper we show practical bioinfor- 
matics experiments that require the facilities of this sort 
of declarative programming language. 

OpenKnowledge is targeted at sharing of detailed 
experimental protocols. By analogy to the current prac- 
tice of writing scripts for combining programs in bioin- 
formatics experiments, we assume that a large scale 
system for sharing experimental protocols would have 
to deal with large numbers of quite specific protocols, 
often designed independently by peers. Effective meth- 
ods of finding, reviewing and repeating experimental 
protocols then become an important component of the 
basic science because without such methods it would be 
unclear what science had been done. 

When conducting experimental studies (or amassing 
information to support experimental studies) from 
Internet sources, each scientist (or group) may adopt a 
variety of roles as information providers, consumers or 
modifiers. Often these roles are narrowly specific, as for 
example the role one adopts when canvassing trusted 
sources for information about specific proteins and 
applying assessment metrics to these that are appropri- 
ate to a particular style of experimentation. In science, 
the roles we adopt and the specific ways in which we 
discharge the obligations of those roles are fundamental 
to establishing peer groups of "like minded" scientists in 
pursuit of related goals by compatible means. The need 
to be precise about such obligations is strongly felt in 
traditional science-hence the use of rigid conventions 
for description of experimental method and monitoring 
of its execution via laboratory notebooks, enabling 
experiments to be monitored, replicated and re-used. 
Analogous structure is beginning to emerge in Internet 
based science. For example the structure of Web service 
composition in Taverna provides a record of the asso- 
ciations between services when using these to manipu- 
late scientific data. Like Taverna, we describe 
interactions. Unlike Taverna, our interaction models are 



part of a system for peer-to-peer communication in 
which specifications of complementary roles in experi- 
mentation are shared between peers as a means of com- 
municating and coordinating experiments. 

The lightweight coordination calculus (LCC) 

An interaction model in LCC is a set of clauses each of 
the form R :: D, where R denotes the role in the interac- 
tion and D is the definition of the role. Roles are of the 
form a(T, P), where T gives the type of role and P is an 
identifier for the individual peer undertaking that role. 
The definition of performance of a role is constructed 
using combinations of the sequence operator (then) or 
choice operator (or) to connect messages and changes 
of role. Messages are either outgoing to another peer in 
a given role (= >) or incoming from another peer in a 
given role (< =). Message input/output or change of role 
can be governed by constraints (connected by the "<— " 
operator) which may be conjunctive or disjunctive. Con- 
straints can be satisfied via shared components regis- 
tered with http://www.openk.org, so that complex 
(possibly interactive) solving methods can be shared 
along with interaction models; or they can be calls to 
services with private data and reasoning methods. Vari- 
ables begin with upper case characters. 

Although it is not shown in the example of Figure 2, 
role definitions in LCC can be recursive and the lan- 
guage supports structured terms in addition to variables 
and constants so that, although its syntax is simple, it 
can represent sophisticated interactions. Notice also that 
role definitions are "stand alone" in the sense that each 
role definition specifies all the information needed to 
complete that role. This means that definitions for roles 
can be distributed across a network of computers and 
(assuming the LCC definition is well engineered) will 
synchronise through message passing while otherwise 
operating independently. Mathching of output messages 
from one peer to input messages of another is achieved 
by simple pattern matching, since (although operating 
independently) the roles were originally defined to work 
together. More sophisticated forms of input/output 
matching have been defined for LCC (to allow for more 
sophisticated ontology matching) but these are not the 



a(r1, A) :: 

ml => a(r2, B) then 
m2 <= a(r2, B). 

a(r2, B) :: 

ml <= a(r1, A) then 
m2 => a(r1, A) <- c. 

Figure 2 A basic interaction model in LCC. 

I J 
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subject of this paper. For a more detailed introduction 
to LCC, see [1]. 

The model in Figure 2 defines an interaction between 
peers in two roles, rl and r2. In the interaction some 
peer, A, in role rl, would send the message, ml, to peer 
B in role r2; then it would wait for a reply from peer B 
with message m2. Conversely, peer, B, in role r2, would 
wait for the message, ml, from peer A in role rl; then it 
would send a reply to peer A with message m2 if it can 
satisfy the constraint c. 

Protein Identifications by MS/MS 

Tandem mass spectrometry (MS/MS) is a technique to 
measure the mass of sample, and has been used for pro- 
tein sequence analysis for more than two decades [4]. A 
number of informatics algorithms have been developed 
to identify protein sequences from the MS spectra with 
various degrees of accuracy for different types of pro- 
teins [5-7]. The basic idea to correlate the spectrum 
data with theoretical sequences derived from known 
sequences and their spectra is complicated by the events 
happened before, during, and after protein synthesis in 
the cells, i.e., pre-, co-, and post-translational modifica- 
tions. As such, MS/MS protein identification techniques 
have some specialisation for handling specific issues. For 
instance, some techniques are better to reduce complex- 
ity of MS spectra and others are better handle modifica- 
tions or mutations. 

The two principal approaches for protein identifica- 
tions by MS /MS include peptide fragment fingerprinting 
(PFF) and de novo sequencing. PFF algorithms, is per- 
formed specifically for candidate peptides extracted 
from a database by building theoretical model spectra 
from theoretical peptides and measuring the similarity 
between the experimental spectra and the modelled 
ones. Most of the search engines, including MASCOT 
[5], OMSSA [6], SEQUEST [7], are available both as 
standalone programs enquiring local copy of public 
genomic-translated databases (GTDB), or as web ser- 
vices connected to online GTDBs. The main drawback 
of this approach is that it can only be used in situations 
where the genome has been sequenced and all predicted 
proteins for the genome are known. This approach is 
not suitable for the proteins with missing post-transla- 
tional modifications (PTMs) and from unsequenced 
genomes. 

The de novo sequencing approach infers knowledge 
about the peptide sequence independently of any infor- 
mation extracted from a pre-existing protein or DNA 
database [8]. Then, the inferred complete or partial 
sequences are compared to theoretical sequences 
according to sequence similarity. It is noted that de 
novo can take a pseudo PFF approach to building a 
pseudo sequence database on the fly: the sequences are 



generated by determining all possible amino acid com- 
positions with a total mass matching the experimental 
precursor mass, and then, for each composition, by 
determining all possible amino acid permutations. Sub- 
sequently, theoretical spectra are computed from the 
pseudo sequences and common peaks between experi- 
mental and theoretical spectrum are counted to find the 
best matched sequences. Some information, e.g., more 
accurate precursor masses and range in the number of 
amino acids, would help reduce the combinational 
complexity. 

For improving the accuracy of the MS/MS protein 
identification, combining knowledge of applicable meth- 
ods is plausible [9]. However, this idea of combining 
methods is yet to take on seriously, let alone to imple- 
ment and validate. And due to the lack of peer perfor- 
mance monitoring in current experimentation platforms, 
it would be difficult to evaluate plausible combinations 
of methods. Thus, a peer ranking facility was implemen- 
ted and experimented under the peer-to-peer Open- 
Knowledge infrastructure for performance monitoring. 
With the peer ranking facility, we also test the feasibility 
of a simple interaction model that combines both PFF 
and de novo methods of MS/MS protein identification. 

Experimentation with peer ranking 

Our peer ranking algorithm for monitoring the simu- 
lated peers follows the style of the PageRank algorithm 
[10]. It works by assigning a ranking to a peer at any 
given time as a function of its previous ranking modified 
by the rankings of each peer with which it has inter- 
acted (we refer below to such peers as supporting 
peers). The modification made by each of these peers is 
spread equally across the set of peers which it supports. 
In the algorithm, we assume that we have available 
minimal information about interactions-only knowing 
for each interaction which peers were involved in it and 
whether it completed successfully. This information is 
less than that we can obtain routinely (since the coordi- 
nating peer has access to all the structure of a com- 
pleted interaction) but we make this simplification to 
make the implementation less domain-dependent and to 
explore how far we can go with the bare minimum of 
data. For proteomic experiments, we implemented an 
even simpler algorithm which should be adequate to 
demonstrate that certain peers are dominant after a per- 
iod of interactions in real-world experiments. 

This study aims to test peer ranking under Open- 
Knowledge infrastructure using simulated experiments 
and real-world bioinformatics experiments. The simu- 
lated experiments involved simulated peers assigned 
with different roles for interactions. The real-world 
bioinformatics experiments were conducted with multi- 
ple peers performing different methods of peptide 
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fragment fingerprinting (PFF) and de novo sequencing 
under OpenKnowledge infrastructure, in which the 
interaction among these peers was coordinated accord- 
ing to an interaction model. 

Methods 

Peer ranking algorithm 

The peer ranking algorithm used for monitoring simu- 
lated peers in this study was based on PageRank [10], 
which notations and formulation were specified and 
implemented as follows: 
Given: 

A data set of initial rankings for each peer, each of the 
form cr(P, T) = R 

A data set of records of successful or failed interac- 
tions, each of the form im(T, I, P x , CP) 

where: 

P is the identifier of a peer 

T is either "pos" (denoting the positive ranking) or 
"neg" (denoting the negative ranking) 

R is the numerical magnitude of the (positive or nega- 
tive) ranking, R p is positive ranking, R n is negative 
ranking 

I is the identifier of the interaction model 
Pi is the initiating peer for an interaction 
CP is the set of peers that subscribe to the interaction 
initiated by Pi 

The algorithm for calculating the current rank of a 
peer was then as follows (where i is the empirically cho- 
sen number of iterations used to obtain stable ranking 
values: 

° For i iterations: 

° For each peer, P: 

■ Calculate rank(P) = (R p , R n ) 

■ assign cr(P, pos) = R p 

■ assign cr(P, neg) = R n 

rank(P) calculates the current rank for peer P, where d 
is an empirically chosen damping value used to tune the 
ranking system (a frequently used value in page ranking 
is 0.85): 

rank (P) = ((1 — d) + d * r (s (pos, P) , pos) , (1 — d) + d * r (s (neg, P) , neg)) 

s(T, P) gives the list of peers supporting peer P. If T is 
"pos" then this is the set of positive support or if T is 
"neg" this is the set of negative support. Note that this 
is a list (which may contain duplicates) rather than a set 
because we are interested in the number of times each 
peer is supported. 

s(T,P) = [Ps | a(P,T,Ps)] 



a(P, T, Ps) is true when peer P is supported by peer Ps 
either positively (if T is "pos") or negatively (if T is 
"neg"). Note that this may (intentionally) generate the 
same instance of Ps more than once. This allows us to 
count the number of times the same peer is supported. 

a (P, T, Ps) if im (T, I, P i; CP) and ((P = Pi and Ps e CP) or (P e CP and Ps = Pi)) 

r(S, T) is the sum of the current ranks for all the peers 
in peer set, S, each peer's rank being divided by the 
number of peers it supports (to apportion the influence 
of the rank evenly across those supported peers). If T is 
"pos" then this is the sum of ranks from positive asso- 
ciations or if T is "neg" this is the sum of ranks from 
negative associations. In other words, r(S, T) is the sum 
for all P in peer set S of cr(P, T)/L, where L is the num- 
ber of peers supported by P. 

Experiments on simulated peers 

The experiments with simulated peers used basic inter- 
action model for data sharing as specified in Figure 3. 
The interaction model described two roles: a data 
source which offers a data set to a data seeker if it 
receives a request for data from the seeker, and a data 
seeker which requests data for a query and caches the 
data received (testing also that the data set was of 
acceptable overall quality). In the interaction model as 
depicted in Figure 3, X is the identifier for the data 
source; Y is the identifier for the data seeker; have_data 
(Q, D) generates the best available set of data, D, known 
to X for query, Q; need_data(Q) generates a data query, 
Q for Y; cache_data(Q, D) merges the data set, D with 
the data cached by Y for query, Q; and acceptable(T, D) 
succeeds when the data set, D, has a mean quality level 
that exceeds the threshold T (note that T is set to a spe- 
cific value). 

This interaction model had no representation of the 
provenance of the data but there was a point in the 
interaction, when the data seeker sent its request to a 
data source, when an appropriate data source needed to 
be chosen. At this point the data seeker could, if it so 
chose, use peer rank information to select the highest 
ranking peer. The rank of a data source peer in this 
interaction, in turn, depended on how frequently the 
data seekers with which it interacted found the quality 
of the data it supplied to be acceptable (otherwise the 
acceptable constraint in the data seeker role failed and 
consequently the interaction overall failed). This gave a 
feedback loop from supply to ranking via the peers that 
requested data. 

To make our simulation as simple as possible we 
represented data quality as a number ranging from 0 
(lowest quality) to 1 (top quality). Instead of storing 
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a(data_source, X) :: 

request(Q) <= a(data_seeker, Y) then 

offer(D) => a(data_seeker, Y) <- have_data(Q, D). 

a(data_seeker, Y) :: 

request(Q) => a(data_source, X) <- need_data(Q) then 
cache_data(Q,D) and acceptable(T, D) <- offer(D) <= a(data_source, X). 



Figure 3 An interaction model for competitive data sharing. 



actual data elements, the peers in our simulation stored 
these quality values (representing the quality of a data 
item). Although in the OpenKnowledge framework it is 
possible for any peer to coordinate interactions, and 
therefore to have any blend of control over service 
orchestration from a highly centralised system to a pure 
peer-to-peer arrangement, in the simulation a single 
peer represented the central database and (to keep it 
even simpler) it generated a set of 10 data elements 
with quality chosen randomly to be between 0 and 1, so 
the data elements that were obtained from it by peers 
tended to have a normal distribution with mean of 0.5. 
We assumed that the quality that was acceptable to 
peers was higher than this mean, setting it at 0.8 for all 
peers, indicating that peers seeking data from the data- 
base obtained a mixture of acceptable (mean quality 
greater than 0.8) and unacceptable results, so the data- 
base built up a negative ranking as well as a positive 
one. 

Experiments on MS/MS protein identification 
Roles and LCC specifications 

Using OpenKnowledge peer-to-peer interaction infra- 
structure and LCC, we built an experimental environ- 
ment to access and manipulate multiple web-enabled 
services or local programs of various types of MS/MS 
sequencing techniques. These services were configured 
in parallel pathways (Figure 4) for alternative execution. 

As depicted in Figure 4, a peer spectra_input, the 
interaction initiator, uploaded the mass spectrum, and 
randomly selected an interaction model to execute. If 
the "de novo + similarity search" interaction model was 
selected, peers assigned to denovo_approach and simi- 
larity_search were invoked. The analysis result was 
passed to peer output_interface for evaluation. Similarly, 
if the PFF analysis interaction model was executed, 
peers responsible for pff_approach were invoked. The 
analysis result was passed to peer output_interface for 
evaluation as well. Evaluation result, success or failure of 
interaction model, was passed to spectra_input in the 
end. 

According to the PFF interaction pathway as specified 
in Figure 5, peer SI uploaded a mass spectrum (Spec) 
from data source, received the evaluation result (Val), 



and terminated the interaction. Peer PFF received a 
mass spectrum from SI, performed PFF analysis of the 
spectrum and forwarded the analysis result Res to peer 
OI which evaluated the result of PFF analysis and then 
passed the evaluation result Val (0 or 1) to peer SI. 

Similarly, in the de novo interaction pathway shown in 
Figure 6, peer SI uploaded a mass spectrum (Spec) from 
data source, received the evaluation result (Val), and 
terminated the interaction. Peer NOVO received a mass 
spectrum from SI, performed de novo analysis of the 
spectrum and forwarded the de novo analysis result 
Denovo to peer SS for similarity database searching. The 
peer OI received the similarity search result Res from 
SS, evaluated the data Res, and then forwarded the eva- 
luation result Val (0 or 1) to peer SI. 

Figure 5 specified the LCC specification for the PFF 
interaction pathway and Figure 6 listed LCC codes for 
the de novo sequencing + database searching pathway. 
Two roles, spectra_input and output _inter face, were spe- 
cified in both interaction pathways with the same argu- 
ments and interactions. Peers subscribed to the roles of 
spectra_input were responsible for uploading MS/MS 
spectra and selecting preferred routes for the spectrum 
interpretation. Peers subscribed to the role output_inter- 
face were in charge of the re-formating, filtering and 
displaying of the final result yielded by the interaction 
model executed. Other roles specified in the two LCC 
interaction models included pff_approach, denovo_ap- 
proach, and similarity _search, which performed PFF, de 
novo sequencing, and database searching, respectively. 
OpenKnowledge components3 

OpenKnowledge Components (OKCs) were developed to 
access and manipulate web servers and/or local programs 
for MS/MS identification, including algorithms OMSSA 
and MASCOT subscribed to the role pff_approach, Pep- 
Novo Win32 Executable [11] and Lutefisk XPvl.O [12] 
performed the role denovo _approach, and MS-BLAST 
[13] subscribed to the role similarity _search. In this 
experiment, parameters for MASCOT was set with data- 
base NCBI nr, enzyme trypsin, and all the other para- 
meters were set to default settings. Similarly, OMSSA was 
run with database nr, enzyme trypsin, maximum missed 
cleavages set as "2", minimum charge to start using multi- 
ply charged products set to "2", all the optional species 
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Figure 4 Peer ranking experiment on MS/MS protein identification. 



being selected, and all the other default parameters for ion 
trap spectrometers. Both Lutefisk and PepNovo were run 
with their default parameters for doubly charged tryptic 
peptides on ion trap MS. MS-BLAST was run with default 
parameters and database nrdb95. 

OKCs for system roles spectra_input and output_inter- 
face were implemented with GUIs for human users to 
upload the MS spectra and display MS /MS identification 



results. The filtering criteria for the results were taken 
from literature [5,6,14] on different MS/MS identifica- 
tion algorithms as listed in Table 1, to achieve a false 
discovery rate (FDR) less than 0.1. 
Experiment execution 

The experiment was based on a benchmark dataset with 
doubly charged tryptic peptides obtained from low- 
energy ion trap LC/MS/MS runs [11]. Each round went 
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a(spectra_input, SI):: 

mass_spectra(SN, Spec) => a(pff_approach, PFF) <- uploadSpec(SN, Spec) 
then 

result(Val) <= a(output_interface, 01) 
then 

null <- Val==1 

a(pff_approach, PFF):: 

mass_spectra(SN, Spec) <= a(spectra_input, SI) 
then 

null <- pff_analysis(SN, Spec, Res) 
then 

results(PFF, Res) => a(output_interface, 01) 

a(output_interface, 01):: 

results(PFF, Res) <= a(pff_approach, PFF) then 
(null <- filter_result1(PFF, Res) 
then 

result(1) => a(spectra_input, SI)) 
or 

rp^nlHfh => a^nprtra innnt 

Figure 5 Interaction pathway for peptide fragement fingerprinting (PFF) 



through only one of the possible routes (Figure 4) with 
one peer to perform the roles specified in the associated 
LCC codes (Figures 5 and 6). 

A single round started with uploading the MS spectra 
data of peptides to the spectra_input peer. In addition 
to spectrum uploading, the OKC developed for this peer 
allowed the peer to select the analysis approach, which 
was either (1) PFF approach or (2) de novo sequencing 
and database similarity searching approach, for MS/MS 
protein identification. In the cases (e.g., the present 
study) when none of the approaches was preferred, the 
system randomly selected one approach. In each route, 
one of the peers subscribed to the role pff_approach, 
denovo_approach, or similarity _search, was randomly 



selected by the system in the experiment. The result of 
protein identification was sent to the peer performing 
the role output_interface for reformatting and filtering 
according to the criteria as shown in Table 1. The fail- 
ure or success message of the execution was finally sent 
to the peer spectra_input in the end of the interaction 
model. The peer ranking for monitoring the perfor- 
mance of the peers was simply based on counting the 
numbers of failures and successes. 

Results and discussion 

The present study obtained results from experiments on 
the basic behaviours of simulated peers monitored by 
PageRank-style peer ranking under OpenKnowledge 



a(spectra_input, SI):: 

mass_spectra(SN, Spec) => a(denovo_approach, NOVO) <- uploadSpec(SN, Spec) 
then 

result(Val) <= a(output_interface, 01) 
then 

null <- Val==1 

a(denovo_approach, NOVO):: 

mass_spectra(SN, Spec) <= a(spectra_input, SI) then 

denovo_result(Denovo_res) => a(similarity_search, SS) <- denovo_analysis(SN, Spec, Denovo_res) 

a(similarity_search, SS):: 

denovo_result(Denovo_res) <= a(denovo_approach, NOVO) then 

results(SS, Sim_res) => a(output_interface, 01) <- similarity_analysis(Denovo_res, Sim_res) 

a(output_interface, 01):: 

results(SS, Res) <= a(similarity_search, SS) then 

(null <- filter_result2(SS, Res) 

then 

result(1) => a(spectra_input, SI)) 
or 

result(O) => a(spectra_input, SI) 



Figure 6 Interaction pathway for de novo sequencing and database searching 
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Table 1 The scores and threshold values for MASCOT, 



OMSSA, and MS-BLAST. 



Peer 


Name of score 


Threshold 


References 


MASCOT 


MASCOT score 


> 30 


Perkins et al., 1999 


OMSSA 


E-value 


< 0.1 


Geer et al., 2004 


MS-BLAST 


MS-BLAST score 


> 57 


Habermann et al., 2004 



The results given scores by MASCOT, OMSSA, MS-BLAST were classified to be 
successes or failures according to their respective thresholds. 



infrastructure and on the feasibility of combining and 
ranking real-world services of peptide fragment finger- 
printing (PFF) and de novo sequencing for MS/MS pro- 
tein identification. 

PageRank-style ranking of simulated peers 

In the experiments on simulated peers, PageRank-style 
ranking under the OpenKnowledge infrastructure 
worked in a manner similar to ranking in the traditional 
Web, where a power law effect distinguishes a few 
dominant pages that obtained high ranks on a topic 
from the majority that retained lower levels of popular- 
ity. This effect encouraged strong competition for popu- 
larity and was a key driver of large scale coordination 
such as the traditional World-wide Web. The tendency 
to have this effect was observed in the interaction model 
given (Figure 3) with populations of peers of increasing 
sizes, including 4, 8, 16, and 32 in the present study. In 
the experiments each peer had identical behaviour and 
each peer was compliant with the protocol described in 
the interaction model, so all interactions run perfectly 
to completion. The results were depicted in the four 
graphs of Figure 7. Each line on the graphs was the 
positive rating of a peer (measured on the y-axis) as it 
changes while the number of interactions increases 
(measured on the x-axis). Each graph shows a single 
peer achieving dominance with a rapid falloff to others 
with lower rank and a "tail" of low ranking peers. As the 
number of peers increased this effect became more pro- 
nounced and, although separation of dominant peers to 
the fullest extent takes more interactions with greater 
peer numbers, the dominant peers began to separate 
early in all cases. 

We also investigated the sensitivity of ranking to 
changes in peer behaviour. For example, whether there 
is any change in ratings if a peer that initially had been 
compliant became less compliant in its behaviour. Fig- 
ure 8 showed the change in positive and negative rating 
for each of four peers. Instead of having been always 
compliant, we made peers sometimes non-compliant by 
making them fail to satisfy the constraint in the interac- 
tion model we introduced above, which would broke the 
protocol. We made the likelihood of failure of a peer 
increased as the peer had more successful interactions, 
and decreased as the peer had more failed interactions, 



so that the likelihood of compliance of each peer alter- 
nated over time. The ratings as shown in Figure 8 chan- 
ged in response to this, as the positive and negative 
rating lines for each peer tended to twine across each 
other. As a peer was successful, its positive rating 
exceeded its negative rating but this success made it 
more likely to fail to be compliant (because we made 
each peer decided to behave less compliantly when it 
had been successful) so its negative rating then climbed 
and exceeded its positive rating, making it less likely to 
be selected by other peers. Overlaid on this twining 
effect was the basic power law effect that we demon- 
strated in Figure 7, so peer 4 had a dominant rating 
(both positive and negative, since its popularity attracted 
many interactions including more that failed as well as 
more that succeeded), followed by peer 3 and then 
(some way behind) by peers 1 and 2. 

This is an interesting finding as in many data sharing 
applications we consider the information about the 
source of the data (e.g., provenance or accuracy) to 
influence which source we use. Traditionally, prove- 
nance was incorporated into data-sharing interactions 
by devising ontologies for provenance and supplying 
task-/domain-specific methods for propagating prove- 
nance information between peers. We could do this in 
the OpenKnowledge infrastructure by writing interac- 
tion models in LCC that specify provenance propagation 
along with the message passing needed for "raw" data 
transmission. This, however, takes engineering effort so 
it is useful if peer ranking could, alone, take some of the 
strain of assigning reputation to peers in a data sharing 
context. 

Initially, the central database would become the most 
popular peer as the other peers had no data so if any of 
them was asked for data the interaction definitely would 
fail, increasing their negative ranking while their positive 
ranking remained unchanged. As the other peers gained 
data from the database through interaction then those 
peers lucky enough to get high quality data could then 
offer it to other peers, raising the positive ranking of the 
supplier and giving the recipient high quality data with 
which it could increase its rating. 

Figure 9 shows the behaviour of this system for nine 
peers interacting with a database. The positive and nega- 
tive ratings for the top-rated peer (indicated on the dia- 
gram) crossed over after about 160 interactions (note 
that the units on the X axis were in tens of interactions) 
because the peer acquired a substantial amount of high 
quality data from the database and was using that to 
dramatically increase its positive rating. Its high rating 
made it attractive to other peers which placed demands 
on it for more data than it possessed so its negative rat- 
ing climbed too, but not enough to overtake its positive 
rating. 
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Figure 10 shows the change in data quality obtained at 
each peer by taking the sum (for the 10 different 
queries) of the mean quality of data obtained for that 
query. The sum would give a possible minimum from 0 
(no data obtained) to 10 (all data for all queries at a 
quality level of 1). After 1000 interactions the sum of 
mean data quality at each peer rises to approximately 6, 
which was consistent with the drop in popularity of the 
central database that we saw in Figure 9. Because peers 
retained high quality data they were able, once they 
eventually obtained high quality data from the central 
database or from another peer, to boost their ratings by 
contributing to more successful interactions than the 
central database (which delivers data a more variable 
quality). The overall effect was to "lift" the data provi- 
sion from the central database into the peer group. 

To show how close our peer to peer method of data 
sharing comes to the ideal situation, where all the peers 
can directly poll the database as frequently as they wish, 
we set up a simulation. In this simulation there was no 
need to rank peers because every peer interacted only 
with the database and the database was assumed to have 
infinite resources so that it could answer every query. 
Figure 10 shows the change in mean quality of data at 
each peer under these ideal conditions. As expected, the 
quality rose quickly to nearly the maximum possible 
because each peer in this simulation simply polled the 
database as many times as it took to get the highest 
quality data. This is ideal but it is impractical when the 
peer group gets large because the database has to deal 
with all the interactions and would quickly become 



overloaded with queries. By contrast, the results of peer 
to peer sharing (Figure 11) showed a slower improve- 
ment in quality but (as seen from Figure 9) this 
improvement was done by querying across the peer 
group rather than risking overload on the central 
database. 

Peer ranking of peers for MS/MS protein identification 

Seven peers interacting in the MS/MS protein identifica- 
tion experiment were allowed to interact according to 
the interaction model (Figures 5 and 6). Each round 
took a randomly selected MS spectra data (number of 
spectra used = 239 different spectra) and recorded the 
ratings and whether the interaction gave a successful 
protein identification results according to the criteria 
shown in Table 1. The final ratings were as shown in 
Table 2. Among all protein identification peers, MS- 
BLAST performed the best according to the final peer 
ranking scores. As MS-BLAST processed the protein 
identification results from two de novo sequencing 
peers, combining de novo sequencing peers (e.g., Pep- 
Novo and Lutefisk) would improve the final ratings. 
Due to the consistently good performance of MASCOT, 
it was selected more frequently than OMSSA and thus 
MASCOT outperformed OMSSA. 

PFF algorithm MASCOT was rated better in peer 
ranking than the two de novo sequencing tools. This 
result was consistent to the previous finding that PFF 
analysis was more accurate than de novo sequencing 
[12]. For de novo sequencing approach, the ratings of 
PepNovo were higher than that of Lutefisk. Even though 



Rating 




Number of interactions (x 10) 
Figure 9 Ratings for data sharing example. 



□ peerl (positive) 

■ peerl (negative) 

□ peer 2 (positive) 

□ peer 2 (negative) 

■ peer 3 (positive) 

□ peer 3 (negative) 

■ peer 4 (positive) 

□ peer 4 (negative) 

■ peer 5 (positive) 

■ peer 5 (negative) 

□ peer 6 (positive) 

□ peer 6 (negative) 

■ peer 7 (positive) 

■ peer 7 (negative) 

■ peer 8 (positive) 

■ peer 8 (negative) 

□ peer 9 (positive) 

□ peer 9 (negative) 

□ database (positive) 

□ database (negative) 
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Sum of mean 
data quality 
per peer 




Number of interactions (x 1 0} 
Figure 10 Ratings for ideal data sharing direct from a single database. 



this experiment did not evaluate how accurate was each 
round of protein identification for each sequence out- 
put, the peer ranking results in this experiment were 
consistent with the performance evaluation result of dif- 
ferent de novo sequencing programs by Frank and Pevz- 
ner [11]. 



There was a directed change of ratings during the 
whole process of interaction as shown in Figure 12. The 
ratings became relatively stable after 100 rounds of 
interaction. The differences among peers became 
obvious after 25 rounds of interaction. The changes in 
ratings followed the behavioural patterns of peer ranking 



Sum of mean 4 
data quality 
per peer 




Number of interactions (x 10) 
Figure 1 1 Ratings for competitive data sharing between peers. 



Leung et al. Automated Experimentation 201 1, 3:3 
http://www.aejournal.net/content/3/1/3 



Page 13 of 14 



Table 2 Peer ranking in MS/MS protein identification experiment. 


Rank 


Peer 


Peer ranking score 


Total runs 


Number of successful runs 


Number of failed runs 


1 


outputjnterface and spectra_input 


0.308 


239 


110 


129 


2 


MS-BLAST 


0.162 


130 


47 


83 


3 


MASCOT 


0.122 


68 


53 


15 


4 


PepNovo 


0.104 


61 


30 


31 


5 


Lutefisk 


0.059 


69 


17 


52 


6 


OMSSA 


0.023 


41 


10 


31 



The interaction with seven peers involved in the PFF and de novo approaches was executed 238 times. The ranking of the peers was based on the number of 
times executed and the numbers of successful and failed interactions. 



as demonstrated in the experiments using simulated 
peers. 

As seen in the experiments of the present study, peer 
ranking is applicable to monitoring peer-to-peer interac- 
tions under the OpenKnowledge infrastructure. With 
more OKCs built for more practical tasks (not limited 
to protein identification) and various performance feed- 
back mechanisms like peer ranking, we would be able to 
include more peers for large-scale experimentation on 
various other issues such as trust [15]. 

Conclusion 

This study demonstrated a use of peer ranking to sup- 
port automated experimentation in open and peer-to- 
peer environments. Both simulated and real-world 
experiments in the present study showed that the Open- 
Knowledge infrastructure with peer ranking capability 



can serve as an evaluative environment for automated 
experimentation. 
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